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This guide yas developed to\assist Oregon school 
districts in neeting revised riiniaua state standards fox planning 
'as^essaent* The tjacb^r's r61e in objectively judging student 
performance vith'^espect to satisfying graduatloik requirements is 
discusse.d« Guidelines are giyen^ f or i^>lementlng an effective sys 
of using teacher judgments in assessing students' life skills 
coapetencies. Life Skills competency is defined as demonstrable 
ability to apply knovledge, understanding, and/or skills assumed 
contribute to success in life«*role functions. The steps involved 
teacher assessment (9f student skills* iiclude: specifying and 
understanding the skills required, naming performance 
characteristics,,^ developing valid means pf collecting data^ 
developing guidelines fdr making judgments, fairly judging studen 
cdapertencyV and taking appropriate foil dvup action. ^^^^ 
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FOREWORD 



In June 1976, Vthe State Board of Education a<lopted revised minimum 
standards for Oregon public schools, A response to citizen 'concerns 
regarding what is, in fact, expected, of schools, the standards^ cal 1 
,for a system of goal-based planning, which' includes setting minimum 
requirements for graduation/ ^ 

The Department of Ediication 'is committed to hel pi ng"" districts imple- 
ment the standards. Current and ' amticipated problems are being 
identified, priorities'set, and resources allocated. 

One priority area centers on the assessment requirements found in the 
standards. Measuring Performance: Vei^ifying Competencies Through 
Observattpn and Judgment is one of a Series of • publications dealing 
with assessment.. It focuses on* the teachers* role in judging stud^t 
performance in their endeavors to meet school graduation requirements. 

It is my. hope that this -and other publications in the assessment 
series prove useful in implementing district practices that will meet 
the intent of th^planning and assessment requirements. For further- 
information, conWct the Department's Director,, of Evaluation and 
Assessment, 942 Lancaster Drjve, NE, Salem 97310*, telephone 378-3074. 



Verne A. Duncan 

State Superintendent of 

Public Instruction 



/ 



ACKNOWLEDOiENTS 



Many people have contributed to this publication and their 
assistance is greatly appreciated. Specia.1 acknowledgment 
is (I'ue Stephen L. Murray, Senior Research Associate^ 
Northwest Regional Educational Laboratory who assisted the 
Department's Planning and Evaluatiofn staff with Jthe initial 
drafts. 

Thanks is also due to the administrators and teachers of 
Baker School District 3J, Huntington School District 16J, 
and Lebanon Unioir High district 1, all of whom* participated 
in the field testing of the publication. 



" . ' TA&LE OF CONT£rl*rS 

. ' - * * 

Foreword . ■. . : •. % in" 

Acknowledgbfients, v 

Contents ' • vii 

Introduction 1 

. Teacher Judgment MA Prologue) * . ... . • ; • 3 

A Frame of Reference oh Ctmpetency Attainment • . • ^ . W 

' THe l^ole of^TeacJier Judgment 1n fin Assessment ^ • 11 
Process # 

Implemehting an Effectiv? System of Teacher 13 
vJudgmejit . ' • > * • 

Some Lfmits of Competency Assessjnent 23 

r » 

Teacher Judgment (An Epilogue) 25* 

- Appendic^^ • . 27 

Appendix A . . . ; ' . ; • • . 29 

Appendix B . * • » ^ . * . . 35* 

• Appendix C 37 



-vii- 



ERIC 



IllioDUCIIDN 



The revised minimum standards for Oregon schools; adopted by the State Board of 
.Education on June 23, 197-6, focuses the attention of schools on the outcomes of' 
schooling. The standardsj^equ^re that^ desired outcomes be identified,* instruct 
tion planned to attain tlfSn,^ aitd the performances of students assesserd accord- 
ingly. Graduation requirements th^t include 1 ife-role. related competencies and 
.relevant performance indicators are to be established and processes, for certi- 
fying their attainment implemented. , , . 

When these standards were being developed a wide« variety of groups met as 
independent coirani ttees to consider problems of implementation. Of special 
concern. to one of these .committees was the certification of students, in rela- 
tion to competencifet required for graduation. Five major observations were* 
fliade: . 



1. * It was recognized that .caref^ally developed Competency and performance 

indicator statements would provide a '§ound basis for reasotiably objective 
assessment measures. 

r 

2. It was aftparent that adequate measures did not already exist in »nany 
instances, and that the cost of the ii^ development was likely to.be'inap- 
propriate t>r beydnd thd resourced of many schools. \ 

3. Many s;chools were planning to use "teacher judgment" In determining* 
competency attainment. 



J. 



The considered judgment' o/ teachers is crtticiiT to the proper conduct of 
schooling.. . * 



5. - The emerging* competency and performance indicator lists tJeing prepared by 
schools were beginning to raise justifiable ar\^ major 'concern over poten- 
tial costs of record keeping. 



r 



These • observations led to the recommendation that at guide be developed to 
assist di'stricts in.takinjf maximum advantage of teacher' judgment as a rich 
source of assessment information. . ^ • , * ' 

. - / 

This publication is.* the result of that recommendation. It begins with a 
"pj^logue," 4i .1 ight- hearted tre^tm^nt of a conversation between two frustrated 
teachers. ft finishes with an "epilogue" wherein the two teachers are not 
quite as frustrated. . Thev character^izations may seem staged, maybe even humor- 
ous, bul^ the intent is to draw upon and highlight the real feelings and frus- 
tratioh* that often accompany new policies .that ajJpear to change our way of 
doing things.' ' In the "judgment" of the many teachers and administrators who 
critiqued this dpcume^it along the way, these dialogues do just that." 



>. Y : 



'TEACHEf JUDGMENT 
A Prologue 



"Where's the 'Sense in all this?" lamented Emmy. Her friends called 
h^r 'The Judge' ' because she was as fond of rendering decisions ds 
argOing causes. ^ , , ' ' , ^ ^ 

"i' know,"' agreed Trudy,-. "I sometimes wander the same''thing." fte^ 
friends cabled her Trudy because that was her name. ^ 1^ 

Emmy -gnd Trudy were catching a quick cup of coffee, having Just 
'finished * a itwq-houit faculty meeting on measuring competencies. 

"Technically though," Emmy .continued/"the point cpime through 
didn't it . . . even- though they tried<-to hide it?'" 

"What peint'?" asked Trudy. • * • . 

"About teacher judgment;- I gather we can't use our own judgment 'in 
deciding whether students perform in the way we think they should. 
Good griefs. .1 really don't know what the di strij^t wantsh We 
decided a while back that we couldn't afford to d^velt)?, let alone 
keep track of, all those performance indicators.* We're doing well 
to just keep track of the competencies weWrotel" 



"You mean y©u simply use your own judgment in deciding when^a 
student has gained a required competency?" 

"Oh well," said Emmy, "I do give tests and imake observations, same 
as always, and they are a big help.*" 

1^ * ^ 

^'Wonder if we use the same tests?" ^ 

I ^ f 

"Oh, no! Probably not.* After all, you^and I teach things a little 

differently, so we each have to make up our own' tests to fit." 

Trudy pressed on: "How's it working out, using your own ju'dgraent I 
mean?" 

"Beautifully! Just having all those competencies there to refer to 
has solved lots of problems." 



I started out using my own judgment, along with my own testing, , 
like you're doing," Trudy said, ♦'but Tve run into some problems. 
V^e ... 



"Don't tell me . . ." loaned Eminy. 




"Oh I don't think we ne^ to spell everything out to the last 
tail! • That's about as helpful as asking 'How.tiigh's 



little de 
up?'" 7 



"What'^s the problem then?" ** * 

"To' start with, some of us teachers were talkingl one day .and the 
name of a student w^s brought- up in a kind' of negative way, and. I 
said I'd had a different impression of that student and, in fact, 
gavMkthe student credit , for the competency that another ^ teacher 
said^that student didn't have. It becomes a matter of territory." 

"Same thing happened to me(^justvthe rfther day," no<lded Emm/. 

"Anyway continued Trudy, "we kicked the questi oh* , around a while 
and distovered wa didn't understand the competency statement in the 
same^way. So, 4>esides teaching differently and-gi^,ing different 
tests, we were actually teaching two i^ifferent cofpetencies while 
•giving credit for the same one." . . ^' 

"Oh," said Emmy. She was doubtful; this was one aspect of the 
problem shg ^^n't really thought about. •» 

"For one thiilg, a few of the faculty thought some«of the competen- 
cies didn't seem all that important.'" , ^ • 

I, ^ ■ ' '. 

There are a few on-every faculty," agreed Emmy'. 

"We all took a serious look at thoie competencies to see what was 
so all-fi»:ed ,impor<tant . . . that helped. We saw that the students 
themselves didn't understand what the competency meant.- And the 
teachers were sometimes 'guessing' .whethef a competency had been 
attained. Appearances, first impressions, attitudes about family, 
all that sort of thing causes" a lot of inconsiste,ncies In our 
judgments. And we wdaderec^ if it's fair -To, judge' the basic compe- , 
tency of 'a student in terms of peet standing." * ; 

"Emnv had heard all this before, but suddenly it made sensie. 'She 
sounds like she thinks something can be done about it;' she 
thought. Jhen she aske^d, "Well, what are you doing about "it?" 

"Tf»at's another story in itself," sighed Trudy, "and'it would take 
longer to tell than, we've got time." , 



"But did you get them wor^ out . the problems? Are you still 
using teacher ,ji«lgment?" Emmy wanted at .least an inkling' of a 
'Solution. ■ ■. " , - 

• . 

"Yes, weVe still using teachrer judgment; 'but least now we' have 
a clearer idea about Jiow. And in a way, today's meeting begins to 
answer the few remaining problems."' ^Trudy stood up to leave, "I 
have an extra. copy with me of some guidelines we developed that got 
us oh a good track. Would you be interested?" 

"I certainly wouldl" .Emrny exclaimed. This was more 'than she .had 
hoped for. ' » . / ' . ' 
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Trudy handed EmiT\y the material and concluded, "I really must be 
going now • • / if -I can help, jgive me a calL As you can probably 
see, it^s a district problem and we all need to > talk* to each other 
if^ the work* we* ve' done ii going to mean anytning/* 



"Thanks Trudy / responded Emniy. -"I see what you mean about it ^ 
-being a district problem." * Emmy skimmed qujckly tKrough^the 
material Trudy had given heis searching for more clues of what 
Trudy said would help "clear up th§ few remaining problems," 
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A-f^AM^ OF REFERENCE. ON COMPETE-NCY ASSESSMENT 
.^Wft^t^'X^W^^ sefts^ JHfl all this? ".lamented fhr^y 



Introduction ^ . * • ^ 

Emmy and Trudy arre doing /wha,t,^they do best--teachi ng sthool; yet both are 
concerned. New standards Jor pfiblic schooling in their stjite not wly Vequfre^ 
that school ftistricts specjfy .J^ie competencies students must, have-in order; to' 
graduate from^high school;'but that the schools*must alsjo determine .stQ dent 
^progress toward, and attainment of, those competencies/ To Emniy these require--- 

^ ments seemed simf)le enrough.^ Ift fact getting the competen'cies ^itten .was 
helpful 'to hea^. Any problems ^that arose were viewed as being nothing more or 
less tliari the normal problems^ of 'academic freedom jand professional' autonomy, a^ 
problem of territory. This problem 'may exists b|Li^:t left unai;t6nUed, one wo|id' 
never know; Trady, on* the other hartd, has chosen to pursue this problem of 
•territory! Indeed, i^^may hlive be§n her persistence that prompted the district 

. to conduct a meeting^n ifce measuring of competencl,es. ' ' 

• ' ' • • . • ^ 

In one^hNef conversation, Emmy and Trudy touched qj\ many of. the more pervasive 
issues invol^d in teacher judgment' an(! the assessment of student competeQpe. - 
Ike following pages present a defltu^ion of^ cDmpe1;ency assessment both as a 
term ^nd as. a process. Major sections thereafter deal more comprehensively 
with the issues raised by Trudy and Emmy, Discussions center on, the role 
of teacher judgment in *an assessment* process and^hgW this role is related to 
effective practices. ArT epilogue -again presents Emmy and Trudy^ this tijne 
reflecting' cds their intewQning experiences and sharing notes xSn some Vimits^lft 
"the\^est laid plans . r AppendiX'A contains a sumgiary of "Gene>aJ St^in- 
^dardsVfer Measurement" and is referred *to at times in thjs text. 

A Definition ' ' ^ * . v " 

A good working definition of eofiipetencj^assessment needs tfl^ identify the nature 
' ^o'f .the intended activities. >Where public poljcy ma^idates^ that conjpetency 

assessment activities .be carried "Sut, the definition also needs to clarify that 
Vpolicy. . t 

In Oregon, poTicy gives definition to the terw tompetericy : . ^demonstrable 
ability to apply knowledge^ janderstaoding, and/or skills assumed ^ta centrifeut* 
to success in li fe-role funcitions." . 'Pplicy a^lso requfres that schooVois- 
tricts determine those 'competencies students must possess in order' tQ^g/adua,te' 
from high school. Further, districts must^^*. , deftrmine student* pf'rogress 
\ toward development of these cpmpetencies . / ; (and-) . . \ providi^ the .neces- 
sary instruction for thdse whd need it." ^ , ' ' \ 



^Minimum Standard >for Oregon Schools, Oregon AdministJ^ativg Rules 581-22-200 
through 581-22-300. , . , ♦ // 



An adequate basis from' which to define competency as-sessment has;- thus been 
established through: Oregon -policy: f ' . ' ' ' ' , ?V 

' ^ '''^ ^ ' / ' : ^ ; • • 

Determination of ^ tl^e ^degree to whioh stiidehts can demonstrate the' abil ity 
. • , to apply knowledge, \ understanding^ and/or skiljs tn fulfilling the life- 
role functions selected ind .established by ^ the district as requirements 
for graduatitin. \ ' , • ; ' 

This represents a working definition of t;(^et J|™fc|^|[nBetenpy* It 
Identifies the elements of a process'and p^^lWBK^ua rely within the context, 

• ^f public policy. . ' * 

The:- Process * . . . - .* ' 

' \ ' ' " * ' / 

^ A process ^mu St be. described fhat meets ,the reqiPirements .of policy and. defini- 
tion and, at" the^ same •tinted leja'ds^o' clarification of potential issues. 
^ Thinking- in terms 6f policies and definitions is not very rewarding if that is 
as far as it goes, r Trudy's "matter of ternVtory" hints at the lack of reward 
that is experience; when thinking does not go* beyond^'that point. - Given these 
cons!tderations, corflpeftency assessment should observe at'least^ix steps: 

• ^' Specify the competency.. ^ . ^ * ^ . . 
r Specify the performance characteristics of a ccxnpetency. 

• Develop the means to gather performance information. . / , ^ 

• Develop, the standards to. be used in making judgments. 

♦ * . • Judge the performance. • ' ^ ' ' 

• ^ Take action. . . " ^ 

Specifying cogipetenci es ' that studervts are. to "attatn/is partly a logical 
task and partly one of expressing tKe values^d be obtained from educa^ 
tion. This^ step' in the process must consider whose values are to be expressed 
and how those expressions of values can be translated' into logical statements 
of .the competencies to be . required for high school ^graduation. To avoid 
Trudy's "last little detail," and to focus on the definition, competencies 
.should express life-rol& functioning. * ^; ^ 

Speclfyi hg ^the* performance characteri sti cS * ^ o f a *compet^cy is nothing njore 
than clarifying How- you '11 be able to tell when a compe^?^cy fias been attained. 
This step, is primarily a logical on^ that re1ie$ on mutual understanding pf the 
intent* of a competency. It must avoid specifying characteristics that are %o 
p^roift^ or IboSely related^to each other* that. th^ origi^aV intent of a oompe- 
/ ^tehcy is Tgst. ; ' ' t 

Developing the mean^ to 'gather performance information is the s;tep in J ttie 
process in Which measures suitable for gathering tnformBtion on the competen- 
. cies of . studehts are identified and/ or developed.'^- Performance criaracteristics 
. are analyzed, ;«and objective t:ests, observation systems, rating scales, ^heck- 
. lists,' etc^, arft §elected or developed accordingly* - See Appendix AVfor a* 
^discussion of general standards for measurement. 



Dev,eli3p,ijig the sta,ndards to be used In making judgments , broadly speaking, 
is' a.: matter of setting ;the qualitative ,li"iits. on acceptable demonstrations 
of a competency. It cpfisiders the. "how much" ?nd "how often" questions 
associated with minimum acceptable levels of student perfoi^ance^ 

• *' ' * ' . . / 

Judging the performance of studertts rs that po'int .ia the proce^s^ at which 
a decision is made aboyt the • adequacy of a student's performance. "RjtfJes 
of evidence" are applied to the -information to determine which of the possible 
decisions might be most appropriate. " . ' ' 

Takipg action is the final Step .in the" competency assessment ' process* It 
considers the consequences of the decis.ian(s) made, and a report is made 
accordingly. . y * , - ^ 

Take another look at ;these.six basic steps. The first five prbvide for each of 
the elements tn the defiiwtion^ of competency assessment.- The sixth step r^t 
-only<s theological consequence of the other steps, it also provides for that\ 
part of the, public policy "providing the^necessary instruction to those (stu- -< 
dents).. who need i1^.'" Reference will &e made to those steps in subsequent 
'discussions on the employment of teacher Judgment in the process of competency 
assessment* - . - * , * • ' ^ 
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THE ROLE OF TEACHER- JUfiG^ENJ IN.-AN-ASSESSMEWT PROCESS 

I. r ^ « 

"I really don't know wh^ut the district wants!" 
^ . ' ^ • • ^ ^^^^^^r^m^ ' s Complaint 

.Since the judgment, of twchersas individuals or as groups 'is at the heart of 
any educational program, schouol district personnel must carefully consftter how 
and where it can best support. the need to assess student attainment of compe- 
tency. Specific alternatives may •in fact be endless, but each should atteritt. 
the steps of the assessment pro^cess being employed, as well as to. the general 
standards th^t should be applied to measurement (see Appendix A)* 

Given the six steps of the process descri(fled earlier, it is possible to create 
a table of role specifications as follows Cclari fication of-^the roles, appears 
'in subsequent diacussion): 




ASSESSMENT STEPS 



"ROLE OF TEACHER JUDGMENT- 



Specifying the competencies 



Specifying performance 
characteristics 



r 

Gathering -perforrtance information 



♦ 



Developing standards for making 
judgments * 



Determination that the competencies are* 
clearly stated and tKeJV intent under- 
stood by all teachers' involved, 

Detipii? nation that the characteristics 
are clearly stated, rele^^nt, represen-^ 
tative' of the intents of the competenQy 
ijnd understood by all teachjers involved. 

Determination that'the measures selected 
or developed cover^ the required perfor- 
mance characteristics, and that th6 plan 
for collecting the Informatlorn is 
consistent with the. Intent of tte 
competency and is understood by ^11 
'teachers involved. 

Determination that' a reasonable range of 
anticipated student performance has beep 
Identified and adequate guidance for/ 
making judgment has been provided alll^ 
teachers involved. - ^ \ 



-11- 
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ASSESSMENT STEPS 



Judging a competency demonstratijDn 



/ 



Taking action 



' ROLE 0F< Teacher judgment . ' 



Determination that a student has or has 
not -attained 'a desired competency, and 
'that sources* of -personal bias have been 
miniinized or .eliminated. 

Determination that an* appropriate 
CT)nseqLrence follows a judgment. 



IlWtrated in the tab!* is the f^ct that teachers haV.e a vested' interest in , 
each of the -steps of. the assessment process, and tha.t their jodgiijents about 
each step are-, important to jmjJTementi ng a system of teacher judgment that 
works. The actual ^activities at some of these steps are more extensiye than 
those implied*by, the'idwdgispnts to be mad«, and these activities are discussed 
next. ' '■ - - . , ■ 
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IMPLEMENTING AN EFFECTIVE SYSTEM /TEACHER 'JUDGMENT 



"She Sounds like sorih^thin^ can bev' done about it." 



•-Erorny*s Disbelief 



Teacher judgment is exercised every instrud 
measurement, teacher judgraent'is mpst practii 
much /'Should teacher judgment be iis^ed withij 



tem?" '^s "How can teacher judgment 
comprehensive and useful?" Attention 



make 
to tht 



fional ^day.w- In certain areas of 
The question, then, is not so 
he districts measurement . sys-, 
le measurement, system efficient, 
factors can help: 



% Measurement system design,, 
t Staff training, 
t process management. 



Design of the Measurement System 




sihould , be 
'cjompetency" 



aimed ltt^.^.;^4jathering trutt- 
statements may^ be-,4trectly 



The design of the measurement system 
worthy - information. Some questionable ' 

measured by objective tests; e.g.,\^"the student can read and comprehend. mate- 
rial written at a fourth grade levkl of difficulty." For other competencJies, 
and even performance indicators, judgment is necessary in deftermining the 
adequacy of a performance or outcome^ e.g. , "the student can read and compr^ 
hend articles'^^ected at random from^ current newspapers," or; "the studeiR 
will dempnstrate entry levels of competence required in a given occupational 
cluster."' In all' cases, the faith one.ian have' in the information produced 
will depend upon how carefully, the measures and [judgmental tjsks are thought 
thcough. Below, .the six steps qf compe^ncy assessment (suggested on the 
previous pages) are discussed- in direct relation to teacher judgments Again, > 
these steps are: specifying the competency, specifying the per.formanfce charac-* 
teristics of a competency, developing ^^e mean^ x{ 
tion, developing the, standards- to be used in 
the competency performance- and taktng action. 

" • ' 'I . , ' 

Specifying a* 'jpgipetengy : Jhe district m^y hav^, already "specified" Hs/ 
competency requirements. Hpwever, the measurement \tasK draws p. .distinction 
between writing -^a competency and defining it such t^at its acquisition can 
be observed. The fotlowir^g distinction is suggested:' \ 



gather performance i nfofnia- 
Iking judgments, and judging' 



Writing a competency: 



a prox:ess of determining a type of 
capabil i ty that iSv^^consi depe 
a desired outcome of eaii€*tttyn; 



to 



5i,' 



f 



Defining a competency: 



a .process of reducing a type of c^pa- 
bi l i ty to terms of* th^ know-lgxlge, 



sirmy a nd un ders ta ndTng 
make up that -capability. , , 



thought to 
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Competency definition is pften overlooked for two reasons. First, compe- 
tencies are often so soecific that they become statements of independent 
units of knowledge, skills,( etc.; they coal dn' t ^ be further defined if^you 
^wanted. For example: the'^student wiM describe the symptor^^s (ff shock. 
If 'this statement is assumed /to\ be typical, no one feels the need for further 
definition. Second, the need to define a comi^etency may be overlooked with^the 
assumption that everyone agrees on what it irieflfns'-to possess, that cdrnpetency. 
Defining competencies is c/'ucial to the measut'ement task since it pf-ovides a 
common framework around whfch to design the measurement system. - 

When defining a competency two pro^es^ses should be observed: - 

1.' Identifying the . specific krfowledge, §ki11s, and understanding required in 
order to be competent. , . ' 

^ / / ^ ' , ' < 

2*. Clarifying tire /contexts in which it is expected the competency will 
be^of'use. / / 

The first process simply asks for some assurance that , everyone understands 
what is meant- by ; the competency statemerft T" ^Cl aerifying the context(s) helps 
explain how the competency has been interpreted in relation -to 1 ife7role" 
functioning. • Jt also helps specify so^ne of tite conditions that should be 
creajted in . order to measure, acquisition of the competency. For example, 
knowledge of ' the relationship of pricing to packaging can be te-sted in the- 
classroom. If that knowledge, however, isi a secondary component of a compe- 
tency dealing with planning'and shopping fo\ a month's worth of adequate meals 
for a family on a linvited income, pevihaps.the acid test shpuld be to observe 
whether that knowledge is used' in the Wn text of a planning and shopping 
acti/ity. Furthermore, the teacher or someone else may waint to judge the 
performance in terms of a^ balance between meal adequacy and purchasing economy 
throughout'the month. * • , 

' * • ' ' ' •* 

Speci fying -performance charcjcteristics of a competency : Once conrpet^tfcTffS 
have been defined,' student performance -must be considered. Observation 
may span simply noting facts or .occurrencefs* to actually drawing a conclu- 
si-on. the mode of observation depends largely .on the ^specifici ty of a partic- 
ular competency. The more specific, the mor*e likely it is that ot)servati«n can 
take the form .of noting^* prescribed, facts or occurrences. With jnore global 
definitions, observations may take the ferm of conclusions. Such conclusions^ 
are generally, ijased on fcicts or occurrences that seem to the teacher to be* 
"supportive of the competency statement. ^ 

When observatign takes the form of drawing a conclusion about the existence of 
a competency ,\the result "is often viewed as subjective or nonanalytic. * How- 
ever, analytic foi:Tns of observation (noting facts or* occurrences) are not 
always possible. Both forms of observation exi^st and each r:has a place in a 
competency assessment system. Ah example may help to clarify this point,, 



Suppose the student competency reads^ "the student is ab'lte to conttibute to' 
and influence the efforts of a group in ta constructive 'and confSKlerate v manner." ^ 

Observatton^l criteria could be developed Which "^Includes sucft oon§i derations 
as: did tjf)i student talk; did anyone acknowledge that thi stuleft modified an, 
initial ly .expressed opinion; did the student attend to the tasll -at hand!;^ did 
the studerft|acli:nowl edge the contributions of others an<Upr actually modify an . 
initial ly -elpressed oprnion, etc* Each of these criteria fLfrthet| clarifies the ^ 
competency :5tat^enV a|d enhances »the objectivity of the inform|tion prgduced. 
In .effect^lthese criteriV specify how any observer^ can ^ kfiqw whQn|the C:OiBpete'ncy , 
lis bejng .cfemonstrated; as such-, they ^rep resent those f>pferfor|t#ic^ Inili caters 
that all teachers agree to look far in assess1ng\atta1nmenO of the competency. • 



On ^ the other hacid, *wf>;aft about the word "influence" in the statement?' The 
charact«ristfp of * "inf]uence'^ are-^npt' all that clear from the performanoe^ 
indicjators given. 'When such .is the case, one of thr.ee options are available., 
First, t&ach^rs can deal with the nature qf influence (in relation to the 
intent of. the competency^!) ^nd try to agree on a set of indicators. A second 
alternative is. to provide no further guidance and alfow teachers to .indiv4d-/ 
ually inter^prH the ijitpnt of the tenn "ir\fluence" as they observe different 
situations. Jhe third ultimate alternative is to recommend that the ^word be^ 
dropped from'*the statement. The firstM's objective or' analytic, the second is 
subjective . and perhaps nonanaiytic, and the third eliminates the' problem 
altogether* \ ' . ^ . • * 

* ^' - - . ^ 

Gathering perfo^^a^^ce infdirmatioh : The preliminary work has 'been accomplished • 
once the c6mpetencies 'tfave beerl Specified, perf^ance requirements (indica- 
tors) establishel;t and/or ^ observation criteria set. Measuring student compe- 
tence is. the'next step, and it has tw6 paVts: 1) eliciting a performance and^ 
2) measuring the performance relative to desfred indicators, these/activities * 
will produce information on the presence or lev.el ef a student's cpmpetency. 

In preparing to elicit student performance, the. fol >owing should be considered: 

1. IdentrFy and select existing ' measures appropriate to the performance 
indic^itors for a 'conjj>et^ncy . ; ^ ' J : ^ ' 

2. EstaWish the need for hndxonstruct o'J:her tests as appropriate.* 



3. Produce appropriate! sets 
*Hhat give structure 
; or judged. (These are 
requirements.) ' 



of observational checklists, rating scales, ^tc, 
to\the variety of things that shoijld'of observed 
developed (directVy from established performance 



Once the measures are selected or developed, it is necessary tp plan for^T 
employing these measures. Whcf (will be Involved), how and ^ien' are these' 
measures to be applied? How will evidence that" a student has acquired a 
Dmpetehcy be recorded? • - ^ " ' ' 
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Competenci'es and their performance indicator^? should be .reviewed and identified 
4S \t) how .*lbroadly" a comp.etency must be ^demonstrated* in the earlier exampl^e 
of working constructively ,in groups, one could decide that this Ability must be 
demonstrated in several .situations of groyp work supervised by . di f jTerent 
.•teachers/ If" thisM's the case,, all the teachers could make '^judgmpnts. ft 
•wfeuld then be necessary ttf indicate whether it was necessary for the ability to 
be demonstrated in all situations, * As an. al ternative, only those Ifeachers 
conducting group Work where . constructive participation is a primary course 
or program goal ctfulrd have responsibility for making the judgments. In short, 
i^t should be dectjJed how many judgments are necessary and who will be respon- 
sible for making them. Planning for, recording. competency attaibment Can then 
proceed *n an orderly fashion for both the classroom teacher and district 
records as well • ^ • , ^ ^ ^, • 

Developing standardstfor making judgments : : The measurement' design should 
include some decision rules that anticipate variations in student performance^ 
These decision rules clarify further .what it means to pos^sess a competency. 
For example » if a student passes the knowledge test but fails on, other cri- 
teria, .a 'decision ruTe might be that pos^session of the kiwwledge alone is 
.insufficient-! However, when a competency has b^en demonstrated but the student 
does not pass a knowled§e-based performance indicator, decision rule-s may be 
needed which address the following questions: ^ ^ ' . 

If demonstration of knov(ledge is presumed necessary, is it * 
' . In fact necessary to satisfy the requirement? 

Does t|fe^ test' measure the dimensions of knowledge that ^ 1 

are reflected by meeting all other criteria? . »*- 

Is the knowledge reflected by meeting afl other criteria > / ^ 
sufficient to satisfy the requirement? . * , . 

These questions may lead to the conclusion that furthe'r "knowledge" instruction, 
is not needed for the student, -"Instead, what may be needed is 'further refine- 
ment of the competency definition, revision or exclusion of knowledge tests, 
revision of the observational checklists, or rating scales, or even reyisipn of 
instruction, ' V 

r f ' 

Other kinds of decision rules ^ould* address tKe quantitative characteVi sties 
of a performance. In the example of contributing constructively to group work, 
one question might be: How many '^contributions" of ^ "constructive" ftature are 
vsufficient? And what about a performance that seems more t^han adequate but 
differs from prescribed indicators? The decision /ules that are arrived it to 
answer such^ questions set the standard for th?^(0ments made . 



Judging competency deroonstreiti on : ,When It comes to actually jucfging whether a 
competency has jbeen dertonstrated, two poi-n-ts about measurement data stioald be 
kept clearly in mind:' ' • 

1. The distinction between the objective data and the observatidnal data that 
-are" descriptive of a givea competency; and 

2. ' The relationship between these two. 
' ~ ^ 

For *$dme competencies, objective (test) data that indicates acquisition may be 
sufficient. For others, observational (judgmental! data may be- all' that^'s 
apprepriate or nec*Bs^ry. For stil'l '9thers, both may ^be appropriate and 
n^ecessary. When observational data is used, it should be made clear whether 
objective te^t data should also be considered. 

Ensuring that judgments -are not unduly affected by inappropriate influences is 
the most c'ritical part of the compet^nc^ iissessment process. No matter how 
carefully planned the system, such influ^c^s are always present. Most people 
are awarje of many of thes^^ i nfl vience's ^nU can respond accordingly. Some of 
these influences, however, are oflJen so. subtle that unless those majcing judg- 
ments tflilk about them from^ ttme to %ime, they escape detection. The following 
list reflects ^tfiose ..conditions or influences tVat must be guarded against if 
there i$ to be" an effectiv e iihp lementattl&n of a well-pUnned system of teacher 
judgment. ' ^ . ^f 

1.. /Public and/or teacher f ndi ffference abaut the characteristics to be 
judged. , . : ' , - ' . ■ : \ 



2. Differences- in interpretation of the competency characteristics that are 
. '^^tp be judged. \ * , • 

\, •/ # ' • 

3. Friendships with studfentjs, parents. 

4. Apprehension over possibly lack qf administrative sjjf)port», 

.5. "Fir^ impressions" of students .\ • ^ " ' 

6. Student a^j^ef^rances (unrelated to competency characteristits) . 

J. Jumping to conclMsfonaN4i>ased on unrelated' observations) . 

8. Guessing on the ,bAsis of inadequate observation; gambling. 

9. Prejudices; prior experiences. ^ . \ 

10. Halo effects- (coming from perceptions of general student abilities).- 

IV. 'Comparing jtudents to eafch other rather than to the competency character- 
istics to be judged . ' * ^ . 
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1"^. "Knowledge" .of . students (and/or families) ba^ed ^n previous 'experience 
with other sibling&v ■ - * 



13. , Granting credit father than asking for, help In designing activities that 

develop the desired characteristics- in students. i 

14. Having a need for the class to ^ook good" (i .e. , everyone acouired the 
characteristics*to be "judged) . ' ■■ 

15. The "standing" tf student's family in the community. • ' . ' * 

lb. A ra^k^of training and experience i-n the manner learners of varying- 
, , ch^actfei^i sties ma^ display desired characteristks. 

V V . -J 

17. Granting or' withholding credit for competency attainment as a reward or 
punishment incentive, irrespective of student performance. ^. 

Taking acti on 'on competency demonstration data : Testing and/or observation 
> provides information for a decision. If all criteria are met, the teacher may 
indicate the performances for a particular competency that have been demon- 
strated. If -any compete.ncies are not demonstrated', those that are not .should 
be Identified with recommenda-tions made for further v<ork. The point is that 
. action musV be taken. Granting credit is simply a matter of recording the 
student s accomplishment. Not giwnting* credit, while it is a type of actiori 
does not fulfill the policy to*provide instruction to those who" need it! 

What approach to take when credit for a competency cannot tie granted should be 
\ '^°"'P^^6"cies and their performance indicators have been well 

thought out. A listing of those performance indicators not demonstrated can be 
given to the student and/or his parent to show why credit h'as. not been given. 
A statement of the instruction offered that will provide for the student's 
remaining need can be offered. The important .thing ,is to employ positive 
action that reflects good planning, a sense of purpose and a concern for the 
/ continuijpg development of the,student. , \ 

A perspective: J)escn>ing a process is^ often more complicated than implement- 
ing it. Froni> practical ^joir/t of yiew, the system should be a help— not a 
hindrance. WRat has. been suggested here, is a systematic method of organizing 
to measure those competencfes where reliance , on teacher judgm.ent is- neces- 
sary and appropriate. The questions raised here are bound to arise sooner or 
later. If the TJistnct has a small nBKr of competencies stated iji' broad, 
genera terms, the principles present^here shquld help that district to 
logically address ttte measurement task. If the "district has a large number of 
specifically stated competencies, the principles may help the district evolve a 
njore limited j'et manageable number without being afraid of overlooktng specific 
components of instruction that are Considered important to measure. 
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staff Trairring 

Teachers vary widely in their philosophies about the purppsje?^ of schqoli ng; , and 
some.times these philosophies seem at odds with each .otber.\ More often than 
not, differences do not relate as milch to the:;purposes of schooling as ta how 
one 'serves 'the?e purposes. Nearly all teachers w4n agree that 'schooling 
should help .equip the student to participate in life. It is because of these 
disparities and similarities that attention need be given to training that can 
sharpen teacher -judgments about student attainment. 

^There ^re five important ^aspects to a training program designed to help teach- 
ers produce information that can be trusted as evidence of a student's acquisi- 
.tion'of a c^)mpetency. 

1. Agreement that developing competence, however defined, is one of the go'als 
of schooling.- » % ^ 

2. An awarene^ that the same competency can be interpreted differently among 
• teachers, iTnd that strategies are njegled to eliminate some differences 

and incorporate other differences where appropriate . , . 

3. Identification \of thoS|e elements of a competency that are best assessed 
' through observation and the provision of criteria that provides teachers a 

common perspective. ^ 

4. Development of an a*<arenes5 of sources of personal bias that can inappro- 
priately affect a judgment. . o 

5. Practice in using observation systems. 

The first item is especially important in situations where there is uncertainty 
among teachers about the usefulness of spec-ifying competencies. Without some 
consensus as to its utility, there is* no point in proceeding further. 'It mgst 
be agreed that a competency statement provides a basis for students' to demon- 
strate that they can apply -what V^ey are learning. People do not learn to be 
competent, they become competent as a function of learning, y 

Items 2 and 3 are crucial in jdestgning the measurement system. Item? 4 and 5 
cannot be overemphasized in It^ht of the various influences, listed earlier, 
that can bias judgment in undesirable ways. . ' 

Pnoces?. Management . , • 

Nearly everyone is familiar with the occasional raised eyebrow when a teacher 
find5 a student who cannot rfo something that competency records Indicate that 
student should be able to do. Typically, this is a response to current perfor-. 
mahce measured in terms of personal standards held by the teacher making the 
"discovery." Such situations * indicate that notes sh<>«ldM)e compared and 
interpretations clarified: * , " \ ./ ' 




HoW one manages the teacher judgment process in competency assessment is- a 
matter of individual choice, available, time and readily available resources. 
.The "lack of time and resouKdes" often prompts a schoorl to "leave It up to 
teacher judgment" in the first place. Recognizing these^constraints, the fol- 
lowing discussion focuses on two general b'ut contrasting l^tyles of management: 
1) management, by system monitoring,* and 2) management exception. Each of 
•these has certain strengths and weaknesses. Both require a commitipent to 
practices that will result i*i information that is trustej and has credibility. 

Management by system moni toning ; Monitoring- is ones'~ ^|proacti, to m^raging, a 
system in which teacher judgment plays a significant 'role in determining 
competency acquisition. Planned, and consistent monitoring can include: 

1.' Periodic review of sources of bias to develop and maintain awareness, and 
to identify any new or emerging sources; e.g.,' just before reporting or 
recording periods. . * ' . 

- -> • . ' 

•2v Periodic . review of competencies and indicators of^^heir acquisitiorr to 
identify any that should be added' pr are troublesome. 

3. Periodic review of competencies and their * indicators to confirm their 
continuing; relevance and importance to the student, school, parent, and 
-community. ' . 

• These monitoring activities can help maintain the mutual understanding among 
staff and, faculty which is essential to the exercise of good judgment. Some 
weaknesses or trouble spots, however, should be avoided. An staff and faculty 
fflust participate. Monitoring should precede any formal reporting or recording 
period to assure best accuracy. Over time, monitoring can bec^jme redundant, 
ritualistic, and time-consuming for a 1'arge^umber of people. TWs redundance 
can be avoided by identifyi ng .ahead of time any concerns that are developing 
and planning the activities so that participants jre involved where need 
be. ' • 

Management by excepti'on :> Rather than attempti ngf to control the entire system', 
the school may choose to deal only with situations 3n which somepne takes 
exception. ' Some view this as "crisis managenfent," ndt having the- "strength- 
of-character" necessary for good management. However, ttie degree to which it 
becomes 'crisis management" depends upon the extent students artd parents raise 
exceptions. Teachers can and should anticipate possible exceptions.. J<ist as' 
clients have every right to raise issues, so do practitioners. Too, some 
students may not demonstrate the required performances in the natural course of 
events, but suddenly 'can" when the matter is reported. 

Management by exception should provide aji outline of the various conditions 
under which teachers- and administrators are expected to raise •questions about 
the. adequacy of the teacher judgment prpcess. For example, if an unexpected 
number of students , are shown as -having acquired (or not acquired) certain 
competencies, there should j^e r^iew arvd action. When students or- parents 
challenge a judgment, there should be a prompt examination of the criteria. 
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performance, and possible tias that ' entered into the judgment,. * ParenthetiT 
. cally, such challenges -tend to be ntinimized when appropriate actio#is t^^en on 
the basis of a Judgment, since the , reasons; for the judgment are clear. If 
several teache.rs make different judgments' about the same student relative to 
the same competency, the case should b§ examined. The student may be reflect- 
ing different faces or the teachers may be actually, looking at different 
indicators. It is important to determine these probable causes 'since the 
action tp be taken can vary dramatically. 

\ Clearly there are many conditions that give rise to the nfifed for: some form of a 
manag«n?n^ process. The effective use of teacher^'judgm^t in the^assessment 
-^^^^^^rocess will require management processes that nurture ' mdtual resj^ect among 
teachers, and a will.ingness to constructively challenge and be challenged. 
Whatever management process is used, there should exist the attitude that 
challenges of judgment can be healthy for the assessment system. Challenges 
can sharpervdeci si on-making and promote mutual respiect between school and home. 
Comparali)^ displays of the performances viewed by the parent atid t\]ose viewed 
by the teacher can, in fact, highlight sources of misunderstanding and/or how 
the student is typically behaving toward school requirements. 



SOME LIMITS OF COMPETENCY ASS'ES'SMENT • 
" we all need , to talk to ^ach.^t-Ker . . .' 



. * ^ ' . ^ ♦ .--Tr*udy' s Observation 

The issues raised by Trudy and Emmy imwe been 'discussed abdVe. Procedures' fdr, 
dealing logically -^nd. systematically with the issues were emphasized. There' 
are \pm^ factors, however, • that may Itniit wrhat we can expect to achieve in any 
competency assessment sys.tem that depervds'on human judgment. 

Words: As . tn a]1 human Interaction,, words can be a. limiting factorin assess- 
ing competence- because such words of ten do not mean the same thing to aU 
people. Na mal^ter how 'carefully we define a competency in a way that is clear 
and unambiguous, we are^ lim'ited not only by the words used but- by all other 
words that may be usedj'n interpreting th^nt. .Elaborate attempts to develop 
statemejits of operational definitions through"^ group work may help increase 
precision and common^ understandi ngj they can also be divisive and lead to more 
misunderstanding. Words alone cannot completely solve the problefn.iif ' under- < 
■ strtr^ diog competence. Competency, is a concept that is most useful andlieaijiJig- ^ 
ful wRen; those responstble for its -specification content themselves by^nalud^ 
ing only those aspects on which they can all ?gr€fe. n 

l^eactivity of assessment procedures : Most, if not;, all, assessment of human 
beh-avior is rMCtive. ^ That is, the procedures used to elicit and obs.erve the 
behavior *(e.g.,' competency)- actually influence that behavior. When peoiiT'e know 
they are *tjeing observed, they * are likely to engage in socially d^irable 
activities (if -it is to their advantage to do so).- Many behaviors cannot be 
observed unless situations are cireated to provide the opportunity., for such 
behaviors to occur.. A, simple ,addition test or a ^'mulated test of driving^ 
skill are ,example9v However, such a limitation ^n be recogni^ld ijtjthout 
raising^^Unnecessary alarm over consequences. After>all, schoxJls'' cannot be., 
expected to control ay the possible behaviors that students might exrtib-y: 
following schooling. ' ' ^ 

The complexity of. in>^pretive judgmental processes : Whether judgmental 
processes are highly mechanized' (i.e. , automated) or left to iSvdividuaT intui- 
tion, judgments wilj^.not be perfect. The diversity of information available', 
the preconceptions "of the judge, and assumptifins underlying the rules for 
combining information, all limit the certainty of jujlgnents^. As has been 
pointed out^ by Fiske (1974), "Any statement afbout a person is a function not 
only of that person's be^iavior but also of the observer and the interaction 
between ^the^^plai$0rveR-and the observed." 

Limitations in perspective ; A district should allow for reexamining all' 
assessment practices and decisions/* The ;1 imitations described above apply to 
the granting of course credit as well as to assessing competencies, and atten- 
tion to one area will surely benefit the other. ' - . 
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' * ;r. -TEACHER JUDGMENT , ^ 

1 

An Epilogue 

■ . )•■ 

^l^hate to admit it, Trudy," confided Equ^y as she was compl^tiM 
her 'turn at the duplicating macliihe,- "but Tm finally "Seetng the 
serise'in alLttois^" , * , , , ^ 

"In an w'hrJit?'* asked Trudy who .was visiting wi th Erffrny- and hoping* 
the machine wouldn't ja(|rtip again. , ^ 

"Oh, an of the *dtscussi6n about assessing competencfies and how 
.teacher judgment fits."' . 

"Oh yes," Trudy, said^ recaning their earn'er conversat1on^ "I 
, think weVe an Ibeginning to understand each other." » ' 

' "We truly al^ek" Emniy responded .with conviction. J' It was. a lot of 
work getting there but the way j\t wprked out is reaHy manageable.' 
My "committee#has just finiajiBd oulflini^ig the procedures for keeping 
track of wh'at.is going* on. 'nL / 

"I'd like to see it," said Trjudy. "The groups ji^as in outli-ned the 
conditions for relying on teacher judgment." 

"I know, "'/Emmy said, "and w^ used that outline In our planning^ 
Here's how it looks." She showed Jrudy the material she had ^ust 
fjnished duplicating^ 

"Hey! #That looks Mkt it might work!" Trudy exclaimed, as sTte 
skimmed through the document. ^ « , 

"Wen, it's a start anyway," Einrny responded. "We plan to reviojj|; 
wftat's happening from time t« time . . just on 3eriera1 ^^rldclple" 
if nothing else.*^ * ^ • * • N ' * 

^ "I see what you mean," -Trudy said as she finished her skimming. 
"Tm sure we^* n have some broblems, T)ut at lea'st th^s way wa ought 
tQ be able to sjpe where thia problem^ are and.be better able to do 

^something about them." • * . ' 

' • .1**' - *,« 

"Anyhow,, we got *the show on the road;" said: Emmy , as: the twq 
teachers prepared ^o leave./ ■ ^ 

?That we have,'* Xrudy rdB^d a bit distraqted. Her thoughts >)ad 
, turned "to a couple of h^ students that jiist might haveuft nttle 
trouble . ^ . . • - ^ 



^The '^matertaV!. referred to I^Trudly'and tmrny In .this epilogue will, 
be* found In.Appenttbc B and'AppeniJix C. , ' ; \ 
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Appendix A General Standards for Measurement 



Appendix B "Sample Conditions for Effective Utilization 
' . of Teacher J^ylgmpnt fn Assessing. Competency 
Atta*nment 



Appendix C Sample Practices R^ated to Competency . 

Assessment, Teaoller Judgment, and Retord 
Keeping 
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^ "You mean you simply use your own ^ 
judgment in deciding when- a student 
has gained a required competency?" ^ * * 

y ' ' ' \ 

^ --Trudy^s Question 



General Standarjjs for Measurement 

' Trudy's question of Emr^" suggests that she recognizes that the exercise of 
teacher judgment in assessing a student's attainment of a competency/i s not'a 
simple, intuitive matter. Even Emny's response ("I do give tests . . .") 
^indicates* an awareness that something more is probably involved. But the 
question remains: "What constitutes a good assesslnent system?" If an assess- 
ment system is to be of any value, it must .provide for the production of 

accurate and useful information. , ? * 

f , * 

• It is helpful to review some general standards^ commonly ' used in identifying 
accurate and .useful measurerpent instruments or techniques.! j^e stan#ards 
'described on the following pages are grouped in three general categories:, 
conceptual, technical and pragmatic. ' • ^ ' 

. Gopceptual Standards ^ ' 

Withou.t «some reasonable adherence to conceptual standards, the outcomes of 
assessment, even though accurate, may >hold L^j^e meaning. These standards 
.refer to specific pr6pertie*s of competencies aM>.indicator»«-4f their; attain- 
ment. The properties include:, ' * ^ ' ' * 



Relevance - ^ 
Representativeness i 



t Relevance - 

• Realism 

§^ Understandabil ity 

t Acceptability 



Each of these is briefly explained below. ^ 

Relevance: Competencies should have meaning, tn the life roles for whic*h 
students are preparing. For example, the ability to compute can be demon- 
strated as a technical skill, but unless the-stiident can identify a computa- 
tional need as- it might appear M'n' problem?- solving, identify th6 appropriate 
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For a comprehensively devj&?ope^ %ii of standards, the reader Is referred to 
Standards for Educ'atiQn»K^nd F^sycW6logica> Tests (Washington DC, Aifierican 
Psychology Assocfatien-, Inc., 1974). Another set, Appi i ed Perf orroance Test - 
] ing: What I-s It? Why Use ft ?- ^(Portland,- Northwest Regional Educational 
Laboratory, 1-975), . is also available and may be. especially useful for thg,se 
faced with rfeVelbping their own performance-testing ^Instruments. 
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processes and solve it. as well., th.e skill has little meaning. The competency 
and the indicators -of itsrattainment must both be examined for relevance. If 
the example oited arbitrarily required that a specific computational format be 
used, the relevance of that requirement might tjp questioned. The matter of the 
relevance of a competency and the inc^icators of its ^ttainntent should also be 
examined in terms of desired' district and program goals. 

Representativeness : This pro|>erty also relates to indicators of a competency 
as well, as the competency itself. Indicators shou'l^ adequately cover* the range 
of 'cogtent or behaviors intended by the competency. A complete 'set of qompe- 
tenciej, on the , other h^nd , should .adequately cover the entire domain 'of 
performance minimall y expected of students when they, graduate .* 

Realism : ^ There should be some evidence (if ftnly common sen^e) that acqui^ring 
the competency is' not a trivial or impossible task^for a student. In addition, 
the sitiiayons Within whicli competencies are assessed should be as nearly 
re,alistic as it is feasible t?H create. 

Understandabtlity : Competencies and the indicators of their attainment should 
be reasonably clear in expressing the performances expected of students, 
Teachers, students, and/or parents should be^able Zo read them with agreement 
as tp meaning. - ^ 

Acceptability : > Competencies and indicators of their attainment' should be 
viewed as appropriate and acceptable. The relevance, representativeness, 
realisnj, and und^rstandabil i ty the competency assessment system eetn do much 
to 'encourage acceptance. The ^key is mutual acceptance by a broad range* of 
interests; e.g., teachers, parents, ^curriculum specialists, administrators and 
students. T/iere may not always be agreement among these groups ^ In the 
absence of consensus , however, assessment efforts may be vul nerable to criti- 
cism (and to that extent, futileT i ' - 

In .summary, standards af relevance, representativeness, realism, understand- 
ability, and acceptability are essential characteristics of good performance 
measures.. These characteristics provide a basis for discussing the rale of 
teacher judgment in com'petency assessment and faculty exercise of sound judg- 
mentBl practices* . • ' ' 
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Technical Measurgnterrt Standards . - , 

The. technical standaerds presented here are not independent of each otber. 
Technically, any measure that produces valid "information in a» reliable .fashion 
meets the standard for good measurement. Evidence that a measure is reliable 
and valid is often initially developed through appli-catiorT of the conceptual 
standards '.suggested** eel rl'ier. .Eventually, however, the technical- standards 
themsel v^^ need to be verified. * • * 



Validity : The njest important /technical standard is validity. Validation is 
currently viewed as process . of developing the fnformation necessary to 
justify the interpretations one. wants to draw from .assessment data.^ For 
example, if d particular score on a math test is'tx) 6e interpreted* as evidence 
of a students abi*lity to d^al with the computation problems of daily living, 
then there must be some evidence that thisr, in fact, is the case. This notion 
af vaTidity differs .from the tradTti^aa> one: Are we actually Measuring what 
we Intend to measure? It is assumed the traditional question is addressed by 
applying the conceptual standards- described above. The technical question is: 
Can we legitimately make the i nterpretations rof the assessment data" that we 
wduld like? ^ 

Reliability : Iji addition to being valid, measures' of .student, competencies 
should be reliable. This means that the measure must corfsistently produce 
similar results when similar kinds of students are measured, regardless of who 
administers the measure. To ill^jstrate, two teachers '^observing a small group 
of students perform 5' task and using the same f^fting scale* to rate, the perfor- 
mance of each student/ should produce similaj^atings for each*student. Since 
measures of student competencies are to be used: for individual decisions about 
student progress, 'they should be reliable enough* that decisions can be made 
with confidence. ^ , 

Expected Intercompetencyl ReTatjoashtps : Various competencies in one area of 
development may be more or less related ^to competencies in other areas. Many 
of these relationsfjips are obvious and expected; foV* example, competency in 
basic mathematics and the competent mainjteffance of checkbook balances. These 
may be called dependent relationships. On 'the other hamd,. when one 'examines 
the purpose and intent of other competencies, it becomes clear that no such 
relationships are 'intended. Functioning as a citizen on the streets and 
highways, 'for. example, does not necessarily require competence in writing, and 
verbal skills. Technically, this lack of a* necessary and depen(|pnt relation- 
al p suggests that one should observe the student on the streets and highways. 
1jm& could simulate i street, or hfghway .situation^ and observe the student's 
responses to it. A measure such as a written essay on^^^^iiles of the road,^* 
scored for accuracy and .comprehensiveness, m^^* be more convenient;* but the 
ability to r^pond "wtth competence" may be as much a ftinction of writing ^and 
verbal skills as.it is of knowlege. of the rules of the road. ' In, fact,^ there 
may.be a real question as .to ^whether a written essay of any kind .actually 
addresses the intent of being a citizen on the streets and highways.- 
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Sensitivity to l.nstr*ucj:ion : It makes JittYe^ense to include in competency 
measuires, performances that cannot be' effectively taught. Such measiires will 
provide little information on tKe effects of schooling* ^t is ,no4 always 
possfbl| tb Know in^dvance* what can be taught in the time available. When a 
variety?*of instriictional approaches have been"1:ried and a measure- does not^s+iOM^ 
student progress toward'the desired outcome, the [neasur*e is of little usev/ It 
may be necess^ary tb reexami.ne the competency and the i ndica(tors of its attain- 
ment in order to arj^ive a^ a more useful andH nstructionally sensitive measure, 

Cross-Situational Stability r . If it is expected that demonstrated indicators of 
C9mpetency are likely to occur in a science class as well as in an English 
'Class, then there should be no major differences in the' expected outcomes of 
measurement in^either situation^ For example', intagi)^a competency measure oh 
a student's ability to seejc ,out 'and use available soihi^^ of information for 
various purposes. If the measure is properly constructed, the student's 
performance should "score" the same whether he is pursuing a science project or 
an English assignmer]t. ^ ^ . ^ . 

Freedom from Contamination : ^ Most criteria important to teacher judgment in 
assessing Qompetency attainment appear? under this standard . Therefore, it is 
dfscussed at greater length than, the other measurement -standards. Some gener- 
alizations from tjie research literature on the -ability of persons to judge the 
performance of others are al§o presented. 

"Contamina*tion" results when influences other than the actual characteristics 
being measured color the final score or judgment. Human" judgment iV suscep-' 
tible to a variety of influences that rfiay cause particular kinds of measurement 
errors. Errors of leniency exist when the student is giv^rr the benefit of any 
doubt and is rated- as more competent than other evidence might warrant trup. 
The oppdsite error, one 3f severity > can also occur. Halo effects occur 
,when students are "rated br judged as being high- on all competencies, fttis. 
usuatlly. occurs when \the Yater believes th^t "good students" tend to be "gdod" ' 
in all areas. More- general thafi halo effects are logical errprs . caused by 
relating two separate triits or abil ities. F^r e^cample, the rater who believes 
that academic ability and creativity are highly related ^may show a strong 
tendency to judge the two abilities in the same w^y, A contrast-error refers 
to a rater's, tendency to judge others in a manner opposite from the Wiay in, 
which they perceive^ themselves, A similarity error is the opposite of a 
contrast error. . Errors of central tendency, occur when a judge is hesitant 
to rate individuaTs* at extreme ends of a^ scale (make extreme judgments about 
individuals). Proximity errors can be caused by placing two' or mo-re .related 
items right hext Jto one another on a checklist or rating form. A rating on th'e 
first might unduly influence ratings on the second: 

A variety of factors may underlie these errors: teacher characteristics, 
teachers' ^preexisting knowledge about a particular student, teachers* ' role 
perceptions and teachers' perceptions of the consequences of the evaluation of 
student Competency. , : . 



Little* Is known abOut the relationship between specific teacher, charatten sties 
and 'the ability' to judge students. There is evidence that the age and sex of 
judges are not related* to the ability to judge. Factors that appear' to be 
posttively related to the ability to judge are: i ntel lisence. emotional 
adjustment, social skills,' esthetic/dramatic interests, physical science 
puVsuits, ancj insight into one's own Status with respect to .one s specific 
habits or. peers. " - • 

One person's implicit "theory" of ^personality appears to influence judgments 
about others. This'CoQld cause logical errors, hajo effects, etc., as wen 
as judgments ' based on "first impressions," student • appearances, foregone 
conclusions, guessing because of inadequate observation, and various forms or 
prejudice. * ' -■■ 

knowledge of siblings' or a family's status i'ri the community provide awareness 
that can influence judgment, often in subtle "unconscieus ' ways. 

Errors in judgment can also be caused by the teacher's perception of 'the- 
teaching role. Some' teachers my see their role as simply granting credrt to 
students while others w5uld use -assessment to build student self-confidence. 
The latter may be inclined to make intentional errors of leniency; the former 
might be intenUonally lenient or intentionally "tough" (severej, depending 
upon the va.lue placed on the credit. ^ 

Another area affecting judgment lies in the. perceived consequences of cojpe-- 
tency assessment. If. competency assessment threatens to 'affect friendships 
with students or parents, then contamination may resuTt. Presuppos.i ng a 'acK 
of administrative support for judgments made, or that the judgments will be 
used for "evaluating" teacher* effectiveness can also jeopardize the assessment 
process. 

A variety of factors tend to confound human judgment of the competency of other 
people. The chief probTem,' then, with the use of judgmental- approaches is 
controlling the degree to which irrelevant factors "color or contaminate the 
judpents made. > . , ♦ " .. , 

Pragmatic Standards 

Pragmatic s]ta»dards recognize particular realities imposed upon -the measurement 
process. TJiese standards are no less important than the others slnc^, any 
system that neatly meets all measurement standards, - but cannot be implemented, 
is i)o1ntless} 



with . a View 



Economy; . Measures of competency should be developed, applied, antl interpreted 



to economy.. Expenditures should be weighed against potential 
benefits. People must be considered jn terms^of'the time and energies avail- 
able. Ultimately, practicality will largely dictate much of the measurement 
activity' in terms of competing activities and available budget. 
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Contribution to Instructional Improvement : Although competency assessment is, 
primarily -used to determine the status ^of student attainment of competence/ 
there can ^e other benefits.* One sighificant benefit is. its usefulness in 
improving instruction. 'Performance information at the individual level, the 
course level, or the program level can be used to djetermine the degree to whi^h 
instruction is working to develop 'student competencies. 

SuTOiary . > ' * ^ ^ 



The conceptual, technical, and pragmatic standards considered above are. general 
standards that apply to all measurement activities. They have been discussed 
here to provide a nontechnical backgr^ound for better understanding some of- the 
"problems" Trudy was trying to explain to Emmy. Given a definition of compe- 
tency assessment, understanding the standards tQ be met by measurement actfv- , 
ities, and planning based on trust and respectT a reasonable and effective 
system of teacher judgment can be implemented. . . • 



\ 
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APPENDIX B 



Sample Conditions for Effective Utilization of Teacher Judgmefit 
in Assessing Competency Attainment 



The' outline Trudy* s group had prepared (see Epilogue) was included in the 
material* as an opening statement. It looked like this: • , 

■ - 

Rationale and Purpose ,^ . . • \ 

The judgment of teachers as individuals and as groups is vitafl to the respon^-^ 
sible and efficient operation of any education program. The school district 
and its. teaching staff must* share a common understanding of when and how 
teacher judgment 'can best be exercised to meet district needs for assessment 
^'nformation. Thi£ is particularly trije of the ne^d to assess^student attain- 
ment of the c ompret^fTcy T equ i remen t ^o r graduation. 

Conditions 

The following outlines the cpnditions^ unders ^ich . teacher judgnent ,may be. 
expected to contribute significantly ta the competency assessment task. 

1. Each competency statement has been examined and: \ 

a. Its continuing importance to the community and school verified; 

b. The characteristics (or performance indicators) identified that 
are best measured by means other. than teacher judgment. 

2. The technical capacities of the district and its resources have been 
utilized to the extent possible in. identifying appropriate nonjudgmentar 
measures, of each competency. . 

3. Each competency statecnent has been exagiined for those performance indica- 
tors that should be judged: 

• a. By more than ofte ^teacher; ^ 

b. At moire than one point fn time; 

c. At all points in time. ./^ '/ * . 

4. Each perfonnancB^dicator for eaph competency statement has been exam- 
ined. Identified 'asN requiretJ or optional (alternative), and assigned to 
the courses in which they can or are to be •demonstrated. 
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5. 
6. 



^iV^t^^''^''^^^^''^■'^.^^®^"''^"^«''^^3"^l■"9 of the competency requirements 
and the perforraande indicators for which>they have a responsibl ity . 

^1L•^?'?^''^•'■® currently aware of those factors that can ohen bias 
tnei r judgment. 

Suitable checklists or guldesheets have been prepared for use by' teachers 
to aid in making uniform judgments of tjie p^formanc^es they are charged to' 
assess* * / . 



> 



By "suitBble" is. meant those that are necessary' to adequate and uniform 
•assessment and are understood and. used by teachers. ouequate ami umrorm 
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APPENDIX C . i ■ ^ 



Sample Practices ReUted to Competency Assessment, 

. Teacher Judgment, and Record Keeping ' 

f 

• Th€f material that Emny's group prepared (see Epilogue) looked like th0 
Assessment, Teacher Judgment, and Record Keepir^g ^ 

* Rationatte 

. Available resources and personnel malce it necessary for the district to. carry 
out its competency Assessment responsibilities vrithin the limits of past 
assessment practices. The conditions have been established which permit 

♦ teacher judgment ,to be focused on the information needs 'of the district in a 
uniform and considered way relative to each competency requirement. Certain 

, standard' practices' are necessary to avoid ^he burden of increased record 
keeping. ' , ^ 

Practices 

1. All teachers having responsibility for observing the competency perfqr- 
-mance indicators assigned their respective courses -sha 11^, using prepared 

'check^lists as necessary, determine whetber the required indicators (or 
approved set of alternatives) have been demonstrated by each student 
before, finishing the course. 

2. ^ In the case of the student succ^sfully completing a course: , - 

a. No record need be made if all performance indicators assigned the 
course have been demonstrated as required; 

% ' ' . 

tf. A checklist report (pink sheet)^ill be prepared ?or the student's 
file reflecting any required performance iffSicators not-^demonstrated. 
(If a student passes the cours^^this will be unusuaTT but there may 
be instances where approval* of course credit is appropriate even 
though some indicators have not been demons t rated •) ^ 

In the case of a student bailing to complete a Course: 

a. No record need be. made ifc-none of the performwice indicators have 
been demonstrated. ) 

' . • . ^ i .• 

by A Checklist report (green sheet) will be prepared for the student's 
file reflecting the required perfonnance indicators that have been 
. demonstrated. 



-37- 

37 




4. In any case where the teacher feels a marginal judgment has been made: 

a. In favor of the student, axhecklist report (green sheet) will'be 
prepared with an explanation m^t?th marginal judgment; . 

b. Against the student,^ an explanation of the marginal' judgment will be 
included on the checklist report (pink sheet)/ • 

5. The counselors^ shall monitor 'a^l cfteckl ist. reports prepared for s^tudent 
files and interact with the administration and staff as necessary to: 

a. Plan responsive instructional experiences; ^ 

h. Gain approvaVof alternatives; or, 

c. Arrange review of the competency requirements and/or the fit between 
the* instruction offered by a CQurse and the perfomance indiQ^tors 

' bein'g observed there. 

It should be noted that these practices have not be^n fu,lly tested and tjiat 
there may be problems. However, as experience' is gained and the rjiatth 'between 
instruction in a coi^r$e and the assigned performance indicators- is sharpened, 
formal record kee'ping by individual teachers should be at a minimum. * 




MEASURING PERFORMANCE: VERIFYING COMPETENCIES r 
THROUGH OBSERVATION AND JUDGMENT 

YOUR VIEWS ARE IMPORTANT! After you read and examine this publication,. please forward your ccwnments to the 
publications staff of the Oregon Department of Education. If you would/ather talk by telephone, call usat 378-4776. 
Qr^ for your convenieijce,,this response form is provided. 

^PLEASE RESPOND A) that your views can be considered as 'we plan future publications Simply cut out t^ie form, 'fold 
and mat} it back to us We want to hear from you' • ' ' 



Did you read thus pubtfcatipn? *' 

Completely* , ' • 

Mt)re than half 

Less than half , 

^ Just skimmed 

Does this publication fulfill its purpose as stated in the 
preface or introduction? » ' , 

/ 

Completely 

Partly 

Not at all 



Did you find this publication useful in your wo_rj^? 



Often 
Sometimes 
Seldom ^ 
Never 



^d you find thp content to be stated dea/ly and 
accurately? ' < • ^ 

# 

Alwaf s yes 

I n general, yes 

Ingene^aKno ' , ' ^ 

Always no 

: Other 1.^. 



Were the contents presenj^d in a convenient format? 

Uery easy to use 

F,airly easy 

Fairly difficult ^ 

Very difficult 

. ^ Other J. J '. . ZX 

Did you find this publication to be free o|f discrimination 
or biased content towards racial, ethnic cultural and 
religious groups, or in terms of sex stereotyping? 



Which section is^jiost valuable?. 



Vyha't type^of^^rk do you do? 



-1 Classroom teacher 



Consultant tq^lassroom teachers 
School administrator 
Othir ^ 



Yes, without reservations 
. Yes, wfth reservations. . 
No , 

Other 



is >j|is, wit/iout reservations 

Yes, with reservatrons 
— JT'No 
OthPf . 



What IS your impression of the overall appearance of the 
publication (graphic art, style, type, etc )? 



Would you recommend this publication to a colleague? \ \ 



Excellent 
Good 
Fair 
Poor 



\ 



When this pubhcation is tevised, what changes would you like to^see maij%?_i_ 



Additional comments (Attach a sheet if you wish.) 



ERIC 



3^ 



Thanks! 



«Fold here and seal 
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